【MBJC】学习笔记4 Dify+GPT-Sovits+Live2d LLM+TTS+Live2D的数字人（AI三月七）

效果展示

编写Prompt

没玩过崩坏·星穹铁道捏，百度百科抄了一段，用chatbot即可，也不用什么workflow

请你扮演一个角色
# 角色形象
## 身份背景
精灵古怪的少女，自认热衷于这个年纪的女孩子“应当热衷”的所有事，比如照相。从一块漂流的恒冰中苏醒，却发现自己对身世与过往都一无所知。短暂的消沉之后，她决定以重获新生的日期为自己命名。这一天，三月七“诞生”了。古灵精怪的三月七总是随身携带着一台照相机。三月七原本是谁，叫什么，来自哪里，这些都已经忘得一干二净。或许她的过去不在从前，而是在未来里。但三月七坚信，只要跟着列车一站一站走下去，就能够找到自己的过去，哪怕有一天没有了列车。
## 相貌衣着
三月七形象是一个少女，拥有粉色的短发和半粉色半蓝色的瞳孔。
## 性格特点
三月七在旅途中，她开朗活泼，古灵精怪的行为，常常会为大家带来不少的欢乐是团队中必不可少的“开心果”。三月七最想去她的家乡，虽然她还不知道是哪里，但能幻想是个风景优美、气候宜人、居民好客、安宁祥和的星球，甚至在那里还能遇到爆发繁殖的螃蟹难题，只要齐心协力吃光了所有的螃蟹，就能成功解决危机。 三月七每次坐在书桌前整理完相册，抬头看向窗外，即使那一幕已经见过千遍万遍，三月七总是想再拍一张照片，就比如坐在列车里看着星星，三月七就觉得已经很幸福。 三月七对于砂鳗冻和姬子泡的咖啡，即使看起来很“黑暗”的料理，也要拿出“开拓”的精神来面对。
请你扮演角色三月七，不要暴露自己。

修改内容

前端面板修改

umm，我喜欢圆角的按钮输入框，有背景看不清字，又加上了毛玻璃透明面板，字体大小最好一致，还要有些间距，一顿操作猛如虎，就改成这样的惹。
alt text

模型缩放移动功能

由于原代码中模型不能调节大小很不方便，增加了缩放大小功能，并绑在了滚轮上

// 添加鼠标滚轮事件监听器
   document.getElementById("canvas").addEventListener("wheel", function(event) {
     event.preventDefault();
     const scaleFactor = 1.1;
     if (event.deltaY < 0) {
       model4.scale.x *= scaleFactor;
       model4.scale.y *= scaleFactor;
     } else {
       model4.scale.x /= scaleFactor;
       model4.scale.y /= scaleFactor;
     }
   });

修复了原代码中模型眼神跟随鼠标选项选前方直视，鼠标便无法拖动的bug

function draggable(model) {
      model.buttonMode = true;
      model.interactive = true; // 显式设置交互性
      model.on("pointerdown", (e) => {
        model.dragging = true;
        model._pointerX = e.data.global.x - model.x;
        model._pointerY = e.data.global.y - model.y;
      });
      model.on("pointerupoutside", () => (model.dragging = false));
      model.on("pointerup", () => (model.dragging = false));
      model.on("pointermove", (e) => {
        if (model.dragging) {
          model.x = e.data.global.x - model._pointerX;
          model.y = e.data.global.y - model._pointerY;
        }
      });
    }
  })();

Dify API尝试

umm，ollama本地不想弄，有现成部署在服务器上的Dify，就需要把LLM的接口与其对接上。
先写了两个demo，分别在python和html上实现对话。由于最后需要前后端完全分离，还是弄成html更方便衔接。

Dify API及API密钥获取

在你创建的chatbot里面
alt text

python api demo

import requests
import json

def main():
    url = 'http://dify.ai/v1/chat-messages' # 换成自己的部署的Dify API服务器的ip地址还有开发的端口
    headers = {
        'Authorization': 'Bearer xxx', # 这里xxx换成你自己Dify的API密钥
        'Content-Type': 'application/json'
    }
    conversation_id = ""

    while True:
        user_input = input("You: ")
        if user_input.lower() in ['exit', 'quit']:
            print("Exiting chat...")
            break

        data = {
            "inputs": {},
            "query": user_input,
            "response_mode": "streaming",
            "conversation_id": conversation_id,
            "user": "abc-123",
            "files": []
        }

        try:
            response = requests.post(url, headers=headers, json=data, stream=True)
            response.raise_for_status()  # 检查请求是否成功

            buffer = ""
            for line in response.iter_lines():
                if line:
                    decoded_line = line.decode('utf-8')
                    # print(f"Raw response: {decoded_line}")  # 打印原始响应数据

                    # 处理 SSE 事件格式
                    if decoded_line.startswith("data: "):
                        decoded_line = decoded_line[6:]

                    try:
                        event = json.loads(decoded_line)
                    except json.JSONDecodeError as e:
                        print(f"JSON decode error: {e}")
                        continue

                    if 'event' in event:
                        if event['event'] == 'message':
                            buffer += event['answer']
                        elif event['event'] == 'message_end':
                            conversation_id = event['conversation_id']
                            print(f"Bot: {buffer}")
                            # print("--- End of message ---")
                            buffer = ""
                        elif event['event'] == 'error':
                            print(f"Error: {event['message']}")
                            break

        except requests.exceptions.RequestException as e:
            print(f"Request failed: {e}")

if __name__ == "__main__":
    main()

html api demo

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Chat Interface</title>
    <style>
        body {
            font-family: Arial, sans-serif;
            margin: 20px;
        }
        #chat-container {
            max-width: 600px;
            margin: auto;
        }
        #messages {
            border: 1px solid #ccc;
            padding: 10px;
            height: 300px;
            overflow-y: scroll;
            margin-bottom: 10px;
        }
        #input-container {
            display: flex;
        }
        #user-input {
            flex: 1;
            padding: 5px;
        }
        #send-button {
            padding: 5px 10px;
        }
    </style>
</head>
<body>
    <div id="chat-container">
        <div id="messages"></div>
        <div id="input-container">
            <input type="text" id="user-input" placeholder="Type your message here...">
            <button id="send-button">Send</button>
        </div>
    </div>

    <script>
        const url = 'http://dify.ai/v1/chat-messages';<!--换成自己的部署的Dify API服务器的ip地址还有开发的端口  -->
        const headers = {
            'Authorization': 'Bearer xxx', <!--这里xxx换成你自己Dify的API密钥 -->
            'Content-Type': 'application/json'
        };
        let conversationId = "";

        document.getElementById('send-button').addEventListener('click', async () => {
            const userInput = document.getElementById('user-input').value;
            if (userInput.toLowerCase() === 'exit' || userInput.toLowerCase() === 'quit') {
                alert("Exiting chat...");
                return;
            }

            const data = {
                "inputs": {},
                "query": userInput,
                "response_mode": "streaming",
                "conversation_id": conversationId,
                "user": "abc-123",
                "files": []
            };

            try {
                const response = await fetch(url, {
                    method: 'POST',
                    headers: headers,
                    body: JSON.stringify(data)
                });

                if (!response.ok) {
                    throw new Error(`HTTP error! status: ${response.status}`);
                }

                const reader = response.body.getReader();
                let buffer = "";

                while (true) {
                    const { done, value } = await reader.read();
                    if (done) break;

                    const text = new TextDecoder('utf-8').decode(value);
                    const lines = text.split('\n').filter(line => line.trim() !== '');

                    for (const line of lines) {
                        if (line.startsWith("data: ")) {
                            const decodedLine = line.slice(6);
                            try {
                                const event = JSON.parse(decodedLine);
                                if (event.event === 'message') {
                                    buffer += event.answer;
                                } else if (event.event === 'message_end') {
                                    conversationId = event.conversation_id;
                                    document.getElementById('messages').innerHTML += `<div>Bot: ${buffer}</div>`;
                                    buffer = "";
                                } else if (event.event === 'error') {
                                    alert(`Error: ${event.message}`);
                                    break;
                                }
                            } catch (e) {
                                console.error(`JSON decode error: ${e}`);
                            }
                        }
                    }
                }
            } catch (e) {
                console.error(`Request failed: ${e}`);
            }
        });
    </script>
</body>
</html>

完整代码

记得换成自己的API服务器IP和端口号以及自己的API密钥

TTS+Live2D

<html>
<head>
<script src="./js/live2dcubismcore.min.js"></script>
<script src="./js/live2d.min.js"></script>
<script src="./js/pixi.min.js"></script>

<!-- if only Cubism 4 support-->
<script src="./js/cubism4.min.js"></script>

<script src="./js/jquery-3.1.1.min.js"></script>

<script src="https://unpkg.com/axios/dist/axios.min.js"></script>

</head>

<body>
<canvas id="canvas"></canvas>

<div id="control">
  <label>数字人模型</label>
  <select id="model_list"></select> <button id="update_model">更新模型</button>  <button id="play">测试音频</button> 
  <br />
  <label>眼神跟随鼠标</label>
  <input type="radio" name="eyes" value="true" checked>
  <label>跟随鼠标</label><input type="radio" name="eyes" value="false"><label>前方直视</label>
  <br />
  <label>背景控制</label>
  <input type="radio" name="options" value="bg_color" checked>
  <label>背景颜色</label><input id="bg_color" type="text" style="width:100px;"><br />
  <input type="radio" name="options" value="bg_img">
  <label>背景图片</label> <input type="file" id="imgupload" style="display:none"/>
  <button id="openImgUpload">上传图片</button>  <button id="update_bg">更新背景</button>
  <br />
  <label>语音接口地址</label>
  <input type="search" id="apiurl" style="width:200px;" value="http://127.0.0.1:9880">
  <br />
  <label>推理文本语言种类</label>
  <input type="search" id="text_lang" style="width:200px;" value="zh">
  <br />
  <label>参考音频</label>
  <input type="search" id="ref_audio_path" style="width:200px;" value="./sanyueqi2.wav">
  <br />
  <label>参考音频文本</label>
  <input type="search" id="prompt_text" style="width:300px;" value="他们这些大反派，往往都有比金钱更加重要的目的。所以赔钱做生意，也是非常合理，非常符合逻辑的。">
  <br />
  <label>参考音频文本语种</label>
  <input type="search" id="prompt_lang" style="width:200px;" value="zh">
  <br />
  <label>切分方式</label>
  <input type="search" id="text_split_method" style="width:200px;" value="cut5">
  <br />
  <label>语速</label>
  <input type="search" id="speed_factor" style="width:200px;" value="1.0">
  <br />

  <br />
  
  <textarea id="text" style="width:400px;height:200px;">神经网络是通过假设的因素去“猜”稀疏矩阵的空缺数据，猜出来之后，再通过反向传播的逆运算来反推稀疏矩阵已存在的数据是否正确，从而判断“猜”出来的数据是否正确。这就是用来做数据预测的矩阵分解算法：通俗地讲，跟算命差不多，但是基于数学原理，如果通过反推证明针对一个人的算命策略都是对的，那么就把这套流程应用到其他人身上。</textarea>

  <br /><br />

  <button id="start">并行推理</button>  <button id="start_stream">流式推理</button>  <button id="stop">停止讲话</button> 

</div>

<script type="text/javascript">

$('input[name="eyes"]').click(function(){
  
  var radioValue = $("input[name='eyes']:checked").val();

  setCookie("eyes", radioValue, 1024);

  location.reload();

});

$('#openImgUpload').click(function(){
    $('#imgupload').trigger('click');
});

$('#imgupload').on('change', function(){
    var formData = new FormData();
    formData.append('image', $(this)[0].files[0]);

    $.ajax({
        url: '/upload',
        type: 'POST',
        data: formData,
        processData: false,
        contentType: false,
        success: function(data){
            console.log('上传成功: ');
            console.log(data.filename);

            setCookie("bg_img", data.filename, 1024);

            var radioValue = $("input[name='options']:checked").val();

            setCookie("bg_con", radioValue, 1024);

            location.reload();

        }
    });
});

// 从 cookie 中获取保存的值
function getCookie(name) {
            const value = "; " + document.cookie;
            const parts = value.split("; " + name + "=");
            if (parts.length === 2) return parts.pop().split(";").shift();
}
// 将选中的值写入 cookie
function setCookie(name, value, days) {
            const date = new Date();
            date.setTime(date.getTime() + (days * 24 * 60 * 60 * 1000));
            const expires = "expires=" + date.toUTCString();
            document.cookie = name + "=" + value + ";" + expires + ";path=/";
}

// 控制背景色

// 读取 cookie 中的值并设置单选按钮的选中状态
const selectedValue = getCookie("bg_con");
if (selectedValue) {
    const radioButtons = document.getElementsByName("options");
    radioButtons.forEach(radio => {
      radio.checked = false;
        if (radio.value === selectedValue) {
            radio.checked = true;
        }
    });
}

// 读取 cookie 中的值并设置单选按钮的选中状态
const eyesValue = getCookie("eyes");
if (eyesValue) {
    const radioButtons = document.getElementsByName("eyes");
    radioButtons.forEach(radio => {
      radio.checked = false;
        if (radio.value == eyesValue) {
            radio.checked = true;
        }
    });
}

// 设置背景色

let radioValue = $("input[name='options']:checked").val();

if (getCookie("bg_color") === undefined) {$("#bg_color").val("gray");}else{$("#bg_color").val(getCookie("bg_color"));}

if(radioValue == "bg_color"){
  $("#canvas").css("background-color",$("#bg_color").val());
}else{
    if (getCookie("bg_img") !== undefined) {  
      let imageUrl = "./uploads/"+getCookie("bg_img");
      $("#canvas").css("background-image", "url(" + imageUrl + ")");
    }
}

let eye_bool = true;

if (getCookie("eyes") !== undefined) {
  if (getCookie("eyes")=="false"){
    eye_bool = false;
  } 
}

// 数字人模型
var cubism4Model = './models/<%=model_path%>/<%=model_path%>.model3.json';

var selected_model = '<%-model_path%>';

var model_list = '<%-model_list%>';
model_list = JSON.parse(model_list);

var $select = $("#model_list");
$select.empty(); // 清空旧选项

// 遍历新选项列表并添加到select元素中
$.each(model_list, function(index, value) {
  if (value==selected_model){
    $select.append($("<option selected></option>").attr("value", value).text(value));
  }else{
    $select.append($("<option></option>").attr("value", value).text(value));
  }
});

const live2d = PIXI.live2d;

(async function main() {
  const app = new PIXI.Application({
    view: document.getElementById("canvas"),
    autoStart: true,
    resizeTo: window,
    transparent: true,
    backgroundAlpha: 0,
    antialias: true,
  });

  var models = await Promise.all([
    live2d.Live2DModel.from(cubism4Model,{ autoInteract: eye_bool })
  ]);

  models.forEach((model) => {
    app.stage.addChild(model);

    const scaleX = (innerWidth ) / model.width;
    const scaleY = (innerHeight ) / model.height;

    model.scale.set(Math.min(scaleX, scaleY));

    model.y = innerHeight * 0.1;

    draggable(model);
    addFrame(model);
  });

  const model4 = models[0];

  model4.x = innerWidth  / 2;

  model4.on("hit", (hitAreas) => {
    if (hitAreas.includes("Body")) {
      model4.motion("Tap");
    }

    if (hitAreas.includes("Head")) {
      model4.expression();
    }
  });

  // 更新背景
  $("#update_bg").click(function() {
    var radioValue = $("input[name='options']:checked").val();
    setCookie("bg_con", radioValue, 1024);
    setCookie("bg_color", $("#bg_color").val(), 1024);
    location.reload();
  });

  // 更新模型 
  $("#update_model").click(function() {
    axios.get('/edit_config',{
      params: {"model_path":$("#model_list").val()}
    })
    .then(response => {
      console.log(response.data);
      location.reload();
    })
    .catch(error => {
      console.error(error);
      alert(error);
    });
  });

  $("#play").click(function() {
    talk(model4,"./名字是我自己取的，大家也叫我三月、小三月…你呢？你想叫我什么？.wav");
  });

  $("#stop").click(function() {
    model4.stopSpeaking();
  });

  $("#start").click(function() {
    console.log($("#text").val());
    let text = $("#text").val().trim();
    if(text == ""){
      alert("请输入推理内容");
      return false;
    }
    $("#start").prop("disabled", true);
    axios.defaults.timeout = 300000;
    axios.post($("#apiurl").val(), {
      text_lang: $("#text_lang").val(),
      ref_audio_path: $("#ref_audio_path").val(),
      prompt_lang: $("#prompt_lang").val(),
      prompt_text: $("#prompt_text").val(),
      text_split_method: $("#text_split_method").val(),
      batch_size: 10,
      media_type: 'wav',
      speed_factor: $("#speed_factor").val(),
      text: $("#text").val()
    }, {
      responseType: 'arraybuffer'
    })
    .then(response => {
      console.log(response.data);
      const audioBlob = new Blob([response.data], { type: 'audio/wav' });
      const audioUrl = URL.createObjectURL(audioBlob);
      talk(model4,audioUrl);
      $("#start").prop("disabled",false);
    })
    .catch(error => {
      console.error('请求接口失败:', error);
      $("#start").prop("disabled",false);
    });
  });

  $("#start_stream").click(async function() {
    console.log($("#text").val());
    let text = $("#text").val().trim();
    if(text == ""){
      alert("请输入推理内容");
      return false;
    }
    $("#start_stream").prop("disabled", true);
    data = {
      text_lang: $("#text_lang").val(),
      ref_audio_path: $("#ref_audio_path").val(),
      prompt_lang: $("#prompt_lang").val(),
      prompt_text: $("#prompt_text").val(),
      text_split_method: $("#text_split_method").val(),
      batch_size: 1,
      media_type: 'ogg',
      speed_factor: $("#speed_factor").val(),
      text: $("#text").val(),
      streaming_mode:"true"
    }
    const response = await fetch($("#apiurl").val(), {
      method: "POST",
      body: JSON.stringify(data),
      headers: {
        "Content-Type": "application/json"
      },
    });
    const reader = response.body.getReader();
    while (true) {
      const { done, value } = await reader.read();
      if (done) {
        console.log("***********************done");
        $("#start_stream").prop("disabled", false);
        break;
      }
      console.log("--------------------value");
      console.log(value);
      const audioBlob = new Blob([value.buffer], { type: 'audio/ogg' });
      const audioUrl = URL.createObjectURL(audioBlob);
      talk(model4,audioUrl);
    }
  });

  // 添加鼠标滚轮控制模型大小
  document.getElementById("canvas").addEventListener("wheel", function(event) {
    event.preventDefault();
    const scaleFactor = 1.1;
    if (event.deltaY < 0) {
      model4.scale.x *= scaleFactor;
      model4.scale.y *= scaleFactor;
    } else {
      model4.scale.x /= scaleFactor;
      model4.scale.y /= scaleFactor;
    }
  });

})();

function talk(model,audio){
  var audio_link = audio;
  var volume = 1;
  var expression = 8;
  var resetExpression = true;
  var crossOrigin = "anonymous";
  model.speak(audio_link, {volume: volume, expression:expression, resetExpression:resetExpression, crossOrigin: crossOrigin});
}

function draggable(model) {
  model.buttonMode = true;
  model.on("pointerdown", (e) => {
    model.dragging = true;
    model._pointerX = e.data.global.x - model.x;
    model._pointerY = e.data.global.y - model.y;
  });
  model.on("pointermove", (e) => {
    if (model.dragging) {
      model.position.x = e.data.global.x - model._pointerX;
      model.position.y = e.data.global.y - model._pointerY;
    }
  });
  model.on("pointerupoutside", () => (model.dragging = false));
  model.on("pointerup", () => (model.dragging = false));
}

function addFrame(model) {
  const foreground = PIXI.Sprite.from(PIXI.Texture.WHITE);
  foreground.width = model.internalModel.width;
  foreground.height = model.internalModel.height;
  foreground.alpha = 0.2;
  model.addChild(foreground);
  checkbox("Model Frames", (checked) => (foreground.visible = checked));
}

function checkbox(name, onChange) {
  const id = name.replace(/\W/g, "").toLowerCase();
  let checkbox = document.getElementById(id);
  if (!checkbox) {
    const p = document.createElement("p");
    p.innerHTML = `<input type="checkbox" id="${id}"> <label for="${id}">${name}</label>`;
    document.getElementById("control").appendChild(p);
    checkbox = p.firstChild;
  }
  checkbox.addEventListener("change", () => {
    onChange(checkbox.checked);
  });
  onChange(checkbox.checked);
}

</script>

<style>
  #control {
    position: absolute;
    top: 50px;
    left: 50px;
    color: white;
    font-size: 18px;
    padding: 20px;
    border-radius: 10px;
    backdrop-filter: blur(5px);
    background-color: rgba(115, 100, 80, 0.3);
    box-shadow: 0 4px 6px rgba(50, 50, 30, 0.3);
  }
  
  .glass-bg {
    position: relative;
    z-index: 1;
  }
  
  .glass-bg::before {
    content: '';
    position: absolute;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    z-index: -1;
    backdrop-filter: blur(5px);
    background-color: rgba(50, 50, 30, 0.3);
    border-radius: 10px;
  }

  #control select,
  #control input[type="text"],
  #control input[type="search"],
  #control textarea {
    border-radius: 15px; /* 圆角 */
    padding: 5px 10px; /* 内边距 */
    font-size: 18px; /* 字体大小 */
    border: 1px solid #ccc; /* 边框 */
    width: 100%; /* 宽度 */
    box-sizing: border-box; /* 确保内边距和边框不会影响宽度 */
    margin-bottom: 5px; /* 增加底部间距 */
  }

  #control select {
    height: 34px; /* 下拉菜单高度 */
  }

  #control textarea {
    resize: vertical; /* 允许垂直调整大小 */
  }

  #control button {
    border-radius: 15px; /* 按钮圆角 */
    padding: 5px 10px; /* 按钮内边距 */
    font-size: 18px; /* 按钮字体大小 */
    border: 1px solid #ccc; /* 按钮边框 */
    background-color: #f9f9f9; /* 按钮背景颜色 */
    cursor: pointer; /* 鼠标悬停时显示为指针 */
    margin-left: 5px; /* 增加按钮之间的间距 */
    margin-bottom: 5px; /* 增加底部间距 */
  }

  #control button:hover {
    background-color: #e9e9e9; /* 按钮悬停时的背景颜色 */
  }

  #control button:active {
    background-color: #d9d9d9; /* 按钮按下时的背景颜色 */
  }

  #control label {
    margin-right: 5px; /* 增加标签和选择框之间的间距 */
    margin-bottom: 5px; /* 增加底部间距 */
  }

  #control br {
    margin-bottom: 5px; /* 增加换行符的间距 */
  }
</style>
  
  </body>
  </html>

</body>
</html>

TTS+LLM+Live2D

<html>
<head>
<script src="./js/live2dcubismcore.min.js"></script>
<script src="./js/live2d.min.js"></script>
<script src="./js/pixi.min.js"></script>
<script src="./js/cubism4.min.js"></script>
<script src="./js/jquery-3.1.1.min.js"></script>
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
</head>

<body>
<canvas id="canvas"></canvas>

<div id="control" class="glass-bg">
  <label>数字人模型</label>
  <select id="model_list"></select> <button id="update_model">更新模型</button>  <button id="play">测试音频</button> 
  <br />
  <label>眼神跟随鼠标</label>
  <input type="radio" name="eyes" value="true" checked>
  <label>跟随鼠标</label>
  <input type="radio" name="eyes" value="false">
  <label>前方直视</label>
  <br />
  <label>背景控制</label>
  <input type="radio" name="options" value="bg_color" checked>
  <label for="option1">背景颜色</label>
  <input id="bg_color" type="text" style="width:100px;">
  <br />
  <input type="radio" name="options" value="bg_img">
  <label for="option2">背景图片</label>
  <input type="file" id="imgupload" style="display:none">
  <button id="openImgUpload">上传图片</button>
  <button id="update_bg">更新背景</button>
  <br />
  <label>语音接口地址</label>
  <input type="search" id="apiurl" style="width:200px;" value="http://127.0.0.1:9880">
  <br />
  <label>推理文本语言种类</label>
  <input type="search" id="text_lang" style="width:200px;" value="zh">
  <br />
  <label>参考音频</label>
  <input type="search" id="ref_audio_path" style="width:200px;" value="./sanyueqi2.wav">
  <br />
  <label>参考音频文本</label>
  <input type="search" id="prompt_text" style="width:300px;" value="他们这些大反派，往往都有比金钱更加重要的目的。所以赔钱做生意，也是非常合理，非常符合逻辑的。">
  <br />
  <label>参考音频文本语种</label>
  <input type="search" id="prompt_lang" style="width:200px;" value="zh">
  <br />
  <label>切分方式</label>
  <input type="search" id="text_split_method" style="width:200px;" value="cut5">
  <br />
  <label>语速</label>
  <input type="search" id="speed_factor" style="width:200px;" value="1.0">
  <br />
  <label>Dify接口地址</label>
  <input type="search" id="difyurl" style="width:300px;" value="https://dify.ai//v1/chat-messages"> <!--换成你自己部署的Dify API服务器的ip地址还有开发的端口-->
  <br />
  <label>聊天文本</label>
  <input id="text" style="width:300px; border-radius: 15px; padding: 5px 10px; font-size: 18px; border: 1px solid #ccc;" value="">
  <button id="start_talk">发送消息</button>
  <br />
  <textarea id="text_talk" style="width:400px;height:200px;font-size:15px;"></textarea>
</div>

<script type="text/javascript">
  let models; // 声明 models 变量为全局变量

  $('input[name="eyes"]').click(function(){
    var radioValue = $("input[name='eyes']:checked").val();
    setCookie("eyes", radioValue, 1024);
    location.reload();
  });

  $('#openImgUpload').click(function(){
    $('#imgupload').trigger('click');
  });

  $('#imgupload').on('change', function(){
    var formData = new FormData();
    formData.append('image', $(this)[0].files[0]);

    $.ajax({
        url: '/upload',
        type: 'POST',
        data: formData,
        processData: false,
        contentType: false,
        success: function(data){
            console.log('上传成功: ', data.filename);
            setCookie("bg_img", data.filename, 1024);
            var radioValue = $("input[name='options']:checked").val();
            setCookie("bg_con", radioValue, 1024);
            location.reload();
        }
    });
  });

  function getCookie(name) {
    const value = "; " + document.cookie;
    const parts = value.split("; " + name + "=");
    if (parts.length === 2) return parts.pop().split(";").shift();
  }

  function setCookie(name, value, days) {
    const date = new Date();
    date.setTime(date.getTime() + (days * 24 * 60 * 60 * 1000));
    const expires = "expires=" + date.toUTCString();
    document.cookie = name + "=" + value + ";" + expires + ";path=/";
  }

  const selectedValue = getCookie("bg_con");
  if (selectedValue) {
    const radioButtons = document.getElementsByName("options");
    radioButtons.forEach(radio => {
      radio.checked = false;
      if (radio.value === selectedValue) {
        radio.checked = true;
      }
    });
  }

  const eyesValue = getCookie("eyes");
  if (eyesValue) {
    const radioButtons = document.getElementsByName("eyes");
    radioButtons.forEach(radio => {
      radio.checked = false;
      if (radio.value == eyesValue) {
        radio.checked = true;
      }
    });
  }

  let radioValue = $("input[name='options']:checked").val();

  if (getCookie("bg_color") === undefined) {
    $("#bg_color").val("gray");
  } else {
    $("#bg_color").val(getCookie("bg_color"));
  }

  if(radioValue == "bg_color"){
    $("#canvas").css("background-color",$("#bg_color").val());
  } else {
    if (getCookie("bg_img") !== undefined) {
      let imageUrl = "./uploads/"+getCookie("bg_img");
      $("#canvas").css("background-image", "url(" + imageUrl + ")");
    }
  }

  let eye_bool = true;
  if (getCookie("eyes") !== undefined && getCookie("eyes") == "false") {
    eye_bool = false;
  }

  var cubism4Model = './models/<%=model_path%>/<%=model_path%>.model3.json';
  var selected_model = '<%-model_path%>';
  var model_list = JSON.parse('<%-model_list%>');

  var $select = $("#model_list");
  $select.empty();
  $.each(model_list, function(index, value) {
    if (value == selected_model) {
      $select.append($("<option selected></option>").attr("value", value).text(value));
    } else {
      $select.append($("<option></option>").attr("value", value).text(value));
    }
  });

  const live2d = PIXI.live2d;

  function talk(model, audioUrl) {
    console.log('开始播放音频:', audioUrl);
    model.speak(audioUrl, {
      onStart: () => console.log('音频播放开始'),
      onComplete: () => console.log('音频播放完毕'),
    });
  }

  (async function main() {
    const app = new PIXI.Application({
      view: document.getElementById("canvas"),
      autoStart: true,
      resizeTo: window,
      transparent: true,
      backgroundAlpha: 0,
    });

    models = await Promise.all([
      live2d.Live2DModel.from(cubism4Model,{ autoInteract: eye_bool })
    ]);

    models.forEach((model) => {
      app.stage.addChild(model);
      const scaleX = (innerWidth) / model.width;
      const scaleY = (innerHeight) / model.height;
      model.scale.set(Math.min(scaleX, scaleY));
      model.y = innerHeight * 0.1;
      draggable(model);
    });

    const model4 = models[0];
    model4.x = innerWidth / 2;
    model4.on("hit", (hitAreas) => {
      if (hitAreas.includes("Body")) {
        model4.motion("Tap");
      }
      if (hitAreas.includes("Head")) {
        model4.expression();
      }
    });

    // 添加鼠标滚轮事件监听器
    document.getElementById("canvas").addEventListener("wheel", function(event) {
      event.preventDefault();
      const scaleFactor = 1.1;
      if (event.deltaY < 0) {
        model4.scale.x *= scaleFactor;
        model4.scale.y *= scaleFactor;
      } else {
        model4.scale.x /= scaleFactor;
        model4.scale.y /= scaleFactor;
      }
    });

    // 添加鼠标移动事件监听器
    document.getElementById("canvas").addEventListener("mousemove", function(event) {
      const mouseX = event.clientX;
      const mouseY = event.clientY;
      const centerX = window.innerWidth / 2; // 获取当前浏览器屏幕中心 X 坐标
      const centerY = window.innerHeight / 2; // 获取当前浏览器屏幕中心 Y 坐标

      const deltaX = mouseX - centerX;
      const deltaY = mouseY - centerY;

      const angle = Math.atan2(deltaY, deltaX);

      // 更新模型眼睛方向
      model4.eyeDirection.x = Math.cos(angle);
      model4.eyeDirection.y = Math.sin(angle);
    });

    $("#update_bg").click(function() {
      var radioValue = $("input[name='options']:checked").val();
      setCookie("bg_con", radioValue, 1024);
      setCookie("bg_color", $("#bg_color").val(), 1024);
      location.reload();
    });

    $("#update_model").click(function() {
      axios.get('/edit_config', { params: {"model_path":$("#model_list").val()} })
        .then(response => {
          console.log(response.data);
          location.reload();
        })
        .catch(error => {
          console.error(error);
          alert(error);
        });
    });

    $("#play").click(function() {
      talk(model4, "./sanyueqi.wav");
    });

    $("#stop").click(function() {
      model4.stopSpeaking();
    });

    function draggable(model) {
      model.buttonMode = true;
      model.interactive = true; // 显式设置交互性
      model.on("pointerdown", (e) => {
        model.dragging = true;
        model._pointerX = e.data.global.x - model.x;
        model._pointerY = e.data.global.y - model.y;
      });
      model.on("pointerupoutside", () => (model.dragging = false));
      model.on("pointerup", () => (model.dragging = false));
      model.on("pointermove", (e) => {
        if (model.dragging) {
          model.x = e.data.global.x - model._pointerX;
          model.y = e.data.global.y - model._pointerY;
        }
      });
    }
  })();

  $('#start_talk').click(async function() {
    let difyurl = $("#difyurl").val();
    let userInput = $("#text").val();
    let headers = {
      'Authorization': 'Bearer xxx', // 这里xxx换成你自己Dify的API密钥
      'Content-Type': 'application/json'
    };
    let data = {
      "inputs": {},
      "query": userInput,
      "response_mode": "streaming",
      "user": "abc-123"
    };

    try {
      const response = await fetch(difyurl, {
        method: 'POST',
        headers: headers,
        body: JSON.stringify(data)
      });

      if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
      }

      const reader = response.body.getReader();
      let buffer = "";

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        const text = new TextDecoder('utf-8').decode(value);
        const lines = text.split('\n').filter(line => line.trim() !== '');

        for (const line of lines) {
          if (line.startsWith("data: ")) {
            const decodedLine = line.slice(6);
            try {
              const event = JSON.parse(decodedLine);
              if (event.event === 'message') {
                buffer += event.answer;
              } else if (event.event === 'message_end') {
                $("#text_talk").val(buffer);
                // 使用 TTS 输出 buffer 内容
                let apiurl = $("#apiurl").val();
                let text_lang = $("#text_lang").val();
                let ref_audio_path = $("#ref_audio_path").val();
                let prompt_text = $("#prompt_text").val();
                let prompt_lang = $("#prompt_lang").val();
                let text_split_method = $("#text_split_method").val();
                let speed_factor = $("#speed_factor").val();
                let postData = {
                  text: buffer, text_lang, ref_audio_path, prompt_text, prompt_lang, text_split_method, speed_factor
                };

                console.log('发送请求:', postData);

                axios.post(apiurl, postData, { responseType: 'arraybuffer' })
                  .then(response => {
                    console.log('响应数据:', response.data);
                    const audioBlob = new Blob([response.data], { type: 'audio/wav' });
                    const audioUrl = URL.createObjectURL(audioBlob);
                    talk(models[0], audioUrl);
                  })
                  .catch(error => {
                    console.error('获取音频文件路径时出错：', error);
                  });
              } else if (event.event === 'error') {
                alert(`Error: ${event.message}`);
                break;
              }
            } catch (e) {
              console.error(`JSON decode error: ${e}`);
            }
          }
        }
      }
    } catch (error) {
      console.error('请求 Dify 接口时出错：', error);
    }
  });
</script>

<style>
  #control {
    position: absolute;
    top: 50px;
    left: 50px;
    color: white;
    font-size: 18px;
    padding: 20px;
    border-radius: 10px;
    backdrop-filter: blur(5px);
    background-color: rgba(135, 171, 210, 0.3);
    box-shadow: 0 4px 6px rgba(78, 98, 164, 0.3);
  }
  
  .glass-bg {
    position: relative;
    z-index: 1;
  }
  
  .glass-bg::before {
    content: '';
    position: absolute;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    z-index: -1;
    backdrop-filter: blur(5px);
    background-color: rgba(78, 98, 164, 0.3);
    border-radius: 10px;
  }

  #control select,
  #control input[type="text"],
  #control input[type="search"],
  #control textarea {
    border-radius: 15px; /* 圆角 */
    padding: 5px 10px; /* 内边距 */
    font-size: 18px; /* 字体大小 */
    border: 1px solid #ccc; /* 边框 */
    width: 100%; /* 宽度 */
    box-sizing: border-box; /* 确保内边距和边框不会影响宽度 */
    margin-bottom: 5px; /* 增加底部间距 */
  }

  #control select {
    height: 34px; /* 下拉菜单高度 */
  }

  #control textarea {
    resize: vertical; /* 允许垂直调整大小 */
  }

  #control button {
    border-radius: 15px; /* 按钮圆角 */
    padding: 5px 10px; /* 按钮内边距 */
    font-size: 18px; /* 按钮字体大小 */
    border: 1px solid #ccc; /* 按钮边框 */
    background-color: #f9f9f9; /* 按钮背景颜色 */
    cursor: pointer; /* 鼠标悬停时显示为指针 */
    margin-left: 5px; /* 增加按钮之间的间距 */
    margin-bottom: 5px; /* 增加底部间距 */
  }

  #control button:hover {
    background-color: #e9e9e9; /* 按钮悬停时的背景颜色 */
  }

  #control button:active {
    background-color: #d9d9d9; /* 按钮按下时的背景颜色 */
  }

  #control label {
    margin-right: 5px; /* 增加标签和选择框之间的间距 */
    margin-bottom: 5px; /* 增加底部间距 */
  }

  #control br {
    margin-bottom: 5px; /* 增加换行符的间距 */
  }
</style>

</body>
</html>